Skip to content

Run the command-line tests three at a time, and check the oracles' mutation score on every change - #391

Merged
ssegura merged 7 commits into
masterfrom
ci/faster-rows
Oct 14, 2026
Merged

ssegura merged 7 commits into
masterfrom
ci/faster-rows

Conversation

@ssegura

@ssegura ssegura commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Related: None. Builds on #390.

What you can do now that you could not before

Nothing a user sees changes. A contributor gets an answer from CI sooner - the command-line tests run
three classes at a time - and a pull request that weakens the oracles' tests now fails CI, because
the mutation score of restest-oracles is checked on every change rather than only when somebody
remembers to run it.

Try it yourself

The mutation score, as one row of CI now runs it:

git fetch && git switch ci/faster-rows
./mvnw --batch-mode --no-transfer-progress -Pmutation verify -pl restest-oracles -am -Djacoco.skip=true

Real output, the lines that matter, in 15 seconds on the maintainer's Mac:

>> Generated 176 mutations Killed 170 (97%)
[INFO] BUILD SUCCESS

The command-line tests, now three classes at a time:

./mvnw verify -pl restest-cli -am
restest-cli's tests, on the maintainer's Mac one process three
wall time 3 min 1 s 1 min 30 s

How it works

Most of the build's time is the command-line tests, which run the whole tool against a stand-in API
and spend their time waiting for it to answer. Each test class has a stand-in of its own on a port of
its own and writes under a directory of its own, and RESTest keeps no global state, so three classes
can run side by side in three processes without touching each other.

Mutation testing changes the oracles' own code on purpose - a comparison flipped, a call removed - and
checks that the tests notice. docs/ci.md said it "costs minutes" and left it to be run by hand; it
takes about ten seconds, so the Linux / Java 25 row now runs it after the build, through a mutation
profile that leaves the ordinary ./mvnw verify as it was.

What changed in the code

  • restest-cli/pom.xml: forkCount 3, reuseForks true, with the reason.
  • restest-oracles/pom.xml: a mutation profile binding PIT's mutationCoverage to verify; the
    comment says it is run on one row of CI.
  • .github/workflows/ci.yml: a step after the build on the ubuntu-latest / Java 25 row; the triggers
    no longer name v2, a branch no commit has reached since 3 October and 76 commits behind master.
  • docs/ci.md: the triggers, the mutation score as a gate, the five production rules its table of
    gates did not name, and four sentences written when the modules were still empty.
  • CONTRIBUTING.md: the sentence that said the mutation threshold is run on demand.

Evidence

  • Tests: none added; ./mvnw verify green locally. CI: this pull request's checks, including the
    new mutation step on the Linux / Java 25 row.
  • Architecture tests: unchanged, none weakened.
  • Mutation score: 97%, 170 of 176 (threshold 85%), unchanged; now checked on every change.
  • Per-request overhead: not touched.
  • Smoke run, two containerised APIs: the smoke job of this pull request.
  • Idle time: not touched.
  • Artefact: the two outputs above.

Decisions taken

Open questions

None.


  • Targets master
  • English throughout, including comments and test names
  • The hard rules in AGENTS.md hold
  • Nothing from "Out of scope" in docs/DESIGN.md without explicit approval

🤖 Generated with Claude Code

ssegura and others added 4 commits October 13, 2026 22:34
… of the clock

A run can now be told to stop once it has sent so many requests: two settings in the schedule
group, `stopAfterRequests` (off) and `requestsBeforeStopping` (1000). On, the loop stops asking
for more at that count and waits for the answers it is owed, as it does at a deadline; the run
ends as it was meant to, not cut short. A run of a known size sends the same requests on a slow
machine as on a fast one, which is what RESTest's own tests of the command line needed: 65 of
their runs were given a second or two of the clock, and twice this week the slowest Windows
machine continuous integration runs on spent that second starting up and answered 3, "nothing was
tested" (AuthCommandTest, on master). Fifteen test classes now give their runs a size; the module
takes 1 min 20 s instead of 3 min.

Two constants of the command move to the same group, by the rule of ADR-0025: how long a run
stopped from outside may spend writing what it found once it has stopped waiting for its answers
(`interruptWritingGrace`, 5 s) and how long it then gives its directory to say what was written
(`interruptLookingGrace`, 1 s). ARunCutShortTest sets the first to a second and the second to ten:
the test took eight seconds and failed on a slow Windows machine whose directory did not answer
within the one second it was given (the other red run on master this week); it now takes one
second and does not depend on the machine.

The settings page, the switches page (a second switch that is not a lever) and the command-line
page say so; ADR-0025 records the four settings. RunLoopTest proves a run of twelve requests sends
twelve, waits for their answers and is not cut short.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ons on one row, and stop watching v2

Three changes to how a build is checked, none to what is checked.

The command-line module's tests run in three processes instead of one (`forkCount` in its POM).
They are the command run end to end, each class against a stand-in API on a port of its own and
writing under a directory of its own, and most of their time is waiting for an API to answer; the
module took three of the four minutes of `./mvnw verify`, and takes a third of that now. Nothing a
run keeps is shared between two processes, which the architecture rules' ban on static mutable
state is there to guarantee.

The mutation score of the oracles is checked on every change, on the `ubuntu-latest` / Java 25
row, as a step after the build. `docs/ci.md` said it "costs minutes" and so was left to be run by
hand, which nobody did unless a pull request touched the oracles; it takes eleven seconds (176
mutations, 97% killed). A `mutation` profile in `restest-oracles/pom.xml` binds the score to that
module's `verify`, so the ordinary build still does not pay for it.

`ci.yml` no longer runs for `v2`, a branch no commit has reached since 3 October and that is
seventy-six behind `master`. The CI page says what runs now, names the five rules its table of
gates had left out, and loses four sentences from the weeks when the modules were empty.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wrapping only: the arguments of the runs given a size, three constructions of the schedule
settings in tests, one test's name and one paragraph of Javadoc. Nothing a test does changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ssegura and others added 2 commits October 14, 2026 18:40
…ed to run three at a time

With three test processes on a runner of four cores, the tests that gave a run one to three seconds
of the clock failed on Linux and Windows: the machine was busy, the second went on starting up, and
the run sent nothing. #390 gives those runs a number of requests instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura
ssegura enabled auto-merge (squash) October 14, 2026 17:13
ssegura added a commit that referenced this pull request Oct 14, 2026
**Related:** None

## What you can do now that you could not before

A release no longer carries an empty `restest-idl` jar, and the repository no longer lists a module
for the IDL language, inter-parameter dependencies, that nothing reads yet. The known limitations now
say plainly that dependencies between parameters are not read, which a user of RESTest 1.x will look
for.

## Try it yourself

```bash
git fetch && git switch chore/idl-module-out-of-the-reactor
./mvnw -q package -DskipTests
dist/assemble.sh target/dist > /dev/null && ls target/dist/restest-*/lib | grep '^restest-'
```

Real output:

```
restest-cli-2.1.1-SNAPSHOT.jar
restest-core-2.1.1-SNAPSHOT.jar
restest-exec-2.1.1-SNAPSHOT.jar
restest-gen-2.1.1-SNAPSHOT.jar
restest-oracles-2.1.1-SNAPSHOT.jar
restest-report-2.1.1-SNAPSHOT.jar
restest-spec-2.1.1-SNAPSHOT.jar
restest-store-2.1.1-SNAPSHOT.jar
```

On `master` the list also has `restest-idl-2.1.1-SNAPSHOT.jar`, whose only class is
`module-info.class`.

## How it works

RESTest is built from modules that may only depend inwards, towards `restest-core`. One of them,
`restest-idl`, was created at the start to hold the IDL language and a constraint solver, and has held
nothing else since. It was still compiled, packaged, copied into every release and listed among what
the tool is made of.

It leaves the build until the first code that reads IDL arrives. The rule that checks the dependencies
between modules keeps a place for it, so it comes back exactly where ADR-0004 put it, checked from its
first class.

## What changed in the code

- Removed: `restest-idl/` and the dependencies of `restest-gen`, `restest-cli` and `restest-arch-tests`
  on it; the two `requires io.restest.idl` in `module-info.java`.
- `ArchitectureRules`: unchanged, with a comment on why the `Idl` layer stays.
- `AGENTS.md`, `docs/DESIGN.md`, `docs/ci.md`, the root and arch-tests `pom.xml` comments: eight
  production modules.
- ADR-0004 and ADR-0010 amended, and the index; `docs/known-limitations.md` says dependencies between
  parameters are not read.

## Evidence

- [x] Tests: none added; `./mvnw verify` green locally. CI: this pull request's checks.
- [x] Architecture tests: unchanged, none weakened; the layered rule keeps its `Idl` layer.
- [x] Mutation score: not applicable.
- [x] Per-request overhead: not applicable.
- [x] Smoke run, two containerised APIs: the smoke job of this pull request.
- [x] Idle time: not applicable.
- [x] Artefact: the list above.

## Decisions taken

- **The module leaves; the rule's layer stays.** Removing the layer would have been a change to an
  architecture test for no gain, and keeping it means nothing has to be decided again when IDL arrives.
- **`AGENTS.md` changes** because it lists the modules; ADR-0030 says such a change is made in a pull
  request, and this is one.
- **Order of merging.** #391, which runs the command-line tests three at a time, edits the same
  paragraph of `docs/ci.md`, and #397, which moves to 2.2.0-SNAPSHOT, changes `restest-idl/pom.xml`;
  whichever is merged second keeps the eight modules and deletes the file.

## Open questions

None.

---

- [x] Targets `master`
- [x] English throughout, including comments and test names
- [x] The hard rules in `AGENTS.md` hold
- [x] Nothing from "Out of scope" in `docs/DESIGN.md` without explicit approval

🤖 Generated with [Claude Code](https://claude.com/claude-code)
…kes a couple of minutes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ssegura
ssegura merged commit 57edcf7 into master Oct 14, 2026
18 checks passed
@ssegura
ssegura deleted the ci/faster-rows branch October 14, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant